Papers with confidence measure
Calibration of Machine Reading Systems at Scale (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to calibrate open setting machine reading systems fail to scale to these settings due to various scale limitations in practical settings. |
| Approach: | They propose to extend existing calibration approaches to calibrate open-domain question answering and claim verification systems to these settings. |
| Outcome: | The proposed calibration methods can selectively predict answers when question answering systems are posed with unanswerable or out-of-the-training distribution questions. |
Improving Open Information Extraction via Iterative Rank-Aware Learning (P19-1)
Copied to clipboard
| Challenge: | Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences. |
| Approach: | They propose an additional binary classification loss to calibrate the extraction likelihood . they propose an iterative learning process where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. |
| Outcome: | Experiments on open information extraction (IE) show that the extraction likelihood is not well calibrated when comparing quality of extracted assertions. |
Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study examines the recognition of irony by humans and automatic systems . a fine-grained annotation scheme allows for improved modeling of ironity in automatic systems. |
| Approach: | They propose a fine-grained annotation scheme that allows for better recognition of irony by humans and automatic systems. |
| Outcome: | The proposed model improves on tweets annotated with high confidence and agreement . it also performs better on high-confidence and highagreement samples compared to automated systems . |